feat(cat-voices): database optimization #3655
Draft
+14,598
−2,138
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* docs: capture initial times * chore: remove cacheDocument * turn off logging * chore: use debugPrint instead of logger * feat: bulk documents save * batching sync * chore: move exact ref resoling to getDocumentData instead of index * fix: DocumentRepository * chore: simplify getting documents data * remove getAllDocumentsRefs and getCachedDocumentsRefs from DocumentRepository. Index is only available in DocumentDataRemoteSource * chore: update docs * simplified document index endpoint * remove randomness from LocalCatGateway * indexing by batch size * wip: filtering by types * Rework filtering refs + checking all refs if already cached in parallel * docs * update indexing.csv * update indexing csv * chore: cleanup * trailing new line * chore * move performance tab to docs * bulk saving typed docs in parallel * chore: revert hardcoded timestamp * chore: typos * split _sync into smaller functions + add documentation * little refactor * fix: analyzer * initial v2 tables * wip * wip * feat: database migration * chore: cleanup * bump batch size * cleanup * chore: remove defensive content decoding * spelling * test on platform * chore: update build scripts * more migration test data * clean up constructors * cleanup * fix: template tests * fix: spelling * fix code-generator earthly target * use logger in migration + wrap in transaction * chore: move migration transaction level higher
* docs: capture initial times * chore: remove cacheDocument * turn off logging * chore: use debugPrint instead of logger * feat: bulk documents save * batching sync * chore: move exact ref resoling to getDocumentData instead of index * fix: DocumentRepository * chore: simplify getting documents data * remove getAllDocumentsRefs and getCachedDocumentsRefs from DocumentRepository. Index is only available in DocumentDataRemoteSource * chore: update docs * simplified document index endpoint * remove randomness from LocalCatGateway * indexing by batch size * wip: filtering by types * Rework filtering refs + checking all refs if already cached in parallel * docs * update indexing.csv * update indexing csv * chore: cleanup * trailing new line * chore * move performance tab to docs * bulk saving typed docs in parallel * chore: revert hardcoded timestamp * chore: typos * split _sync into smaller functions + add documentation * little refactor * fix: analyzer * initial v2 tables * wip * wip * feat: database migration * chore: cleanup * bump batch size * cleanup * chore: remove defensive content decoding * chore: daos * spelling * saveAll * test on platform * chore: update build scripts * feat: DocumentsV2Dao methods * simple proposals pagination query * chore: create a JoinedProposalBriefEntity * rename method * feat: exclude hidden proposals * more tests * renaming and splitting logic into smaller parts * feat: per language strategy * remove CatalystDatabaseLanguage in favor of raw queries as they are easier to mange * remove Index Strategy Documentation * handle case where ref is empty * migration now includes indexes * use v2 documents table for saveAll and isCachedBulk * adds ActionType to JoinedProposalBriefEntity * adds versionIds to JoinedProposalBriefEntity * comments count * adds isFavorite to JoinedProposalBriefEntity * add template to JoinedProposalBriefEntity * adds documentsLocalMetadata table for auto updates * Update docs * use v2 proposals query for discovery most recent section * feat: simplify most recent proposals section * add proposal fav status for v2 tables * local proposal fav status update * update fav state locally for faster feedback * self review * fix tests * update times * fix: analyzer * more migration test data * clean up constructors * cleanup * fix: template tests * spelling * fix: spelling * chore: PR review adjustments * fix: add discovery specific colors (#3637) * fix code-generator earthly target * use logger in migration + wrap in transaction --------- Co-authored-by: Ryszard Schossler <[email protected]>
damian-molinski
added this to the
M5: Voting & Delegation - Production Readiness milestone
…base_optimization_3528
damian-molinski
added
the
no-track
* docs: capture initial times * chore: remove cacheDocument * turn off logging * chore: use debugPrint instead of logger * feat: bulk documents save * batching sync * chore: move exact ref resoling to getDocumentData instead of index * fix: DocumentRepository * chore: simplify getting documents data * remove getAllDocumentsRefs and getCachedDocumentsRefs from DocumentRepository. Index is only available in DocumentDataRemoteSource * chore: update docs * simplified document index endpoint * remove randomness from LocalCatGateway * indexing by batch size * wip: filtering by types * Rework filtering refs + checking all refs if already cached in parallel * docs * update indexing.csv * update indexing csv * chore: cleanup * trailing new line * chore * move performance tab to docs * bulk saving typed docs in parallel * chore: revert hardcoded timestamp * chore: typos * split _sync into smaller functions + add documentation * little refactor * fix: analyzer * initial v2 tables * wip * wip * feat: database migration * chore: cleanup * bump batch size * cleanup * chore: remove defensive content decoding * chore: daos * spelling * saveAll * test on platform * chore: update build scripts * feat: DocumentsV2Dao methods * simple proposals pagination query * chore: create a JoinedProposalBriefEntity * rename method * feat: exclude hidden proposals * more tests * renaming and splitting logic into smaller parts * feat: per language strategy * remove CatalystDatabaseLanguage in favor of raw queries as they are easier to mange * remove Index Strategy Documentation * handle case where ref is empty * migration now includes indexes * use v2 documents table for saveAll and isCachedBulk * adds ActionType to JoinedProposalBriefEntity * adds versionIds to JoinedProposalBriefEntity * comments count * adds isFavorite to JoinedProposalBriefEntity * add template to JoinedProposalBriefEntity * adds documentsLocalMetadata table for auto updates * Update docs * use v2 proposals query for discovery most recent section * feat: simplify most recent proposals section * add proposal fav status for v2 tables * local proposal fav status update * update fav state locally for faster feedback * self review * fix tests * update times * fix: analyzer * more migration test data * clean up constructors * cleanup * fix: template tests * spelling * fix: spelling * chore: PR review adjustments * add order parameter * ProposalsOrder docs * GetProposalsBriefPage supports order * add filters object * proposalsBriefPage filtering * fix: status filtering * more draft proposals filtering tests * Campaign proposals filter * update docs * expose getVisibleProposalsCount and tests * expose filters parameter * integrate proposals page with v2 queries * chore: increase time diff between proposals * chore: reduce count query tables watched when not needed * local proposals cubit * local proposal fav staus update * docs * fix: add discovery specific colors (#3637) * fix code-generator earthly target * use logger in migration + wrap in transaction * spelling * rename category to categoryId for better consistency * proposals per tab selector * release completed in close * extract early return logic into function --------- Co-authored-by: Ryszard Schossler <[email protected]>
…base_optimization_3528
…base_optimization_3528
* docs: capture initial times * chore: remove cacheDocument * turn off logging * chore: use debugPrint instead of logger * feat: bulk documents save * batching sync * chore: move exact ref resoling to getDocumentData instead of index * fix: DocumentRepository * chore: simplify getting documents data * remove getAllDocumentsRefs and getCachedDocumentsRefs from DocumentRepository. Index is only available in DocumentDataRemoteSource * chore: update docs * simplified document index endpoint * remove randomness from LocalCatGateway * indexing by batch size * wip: filtering by types * Rework filtering refs + checking all refs if already cached in parallel * docs * update indexing.csv * update indexing csv * chore: cleanup * trailing new line * chore * move performance tab to docs * bulk saving typed docs in parallel * chore: revert hardcoded timestamp * chore: typos * split _sync into smaller functions + add documentation * little refactor * fix: analyzer * initial v2 tables * wip * wip * feat: database migration * chore: cleanup * bump batch size * cleanup * chore: remove defensive content decoding * chore: daos * spelling * saveAll * test on platform * chore: update build scripts * feat: DocumentsV2Dao methods * simple proposals pagination query * chore: create a JoinedProposalBriefEntity * rename method * feat: exclude hidden proposals * more tests * renaming and splitting logic into smaller parts * feat: per language strategy * remove CatalystDatabaseLanguage in favor of raw queries as they are easier to mange * remove Index Strategy Documentation * handle case where ref is empty * migration now includes indexes * use v2 documents table for saveAll and isCachedBulk * adds ActionType to JoinedProposalBriefEntity * adds versionIds to JoinedProposalBriefEntity * comments count * adds isFavorite to JoinedProposalBriefEntity * add template to JoinedProposalBriefEntity * adds documentsLocalMetadata table for auto updates * Update docs * use v2 proposals query for discovery most recent section * feat: simplify most recent proposals section * add proposal fav status for v2 tables * local proposal fav status update * update fav state locally for faster feedback * self review * fix tests * update times * fix: analyzer * more migration test data * clean up constructors * cleanup * fix: template tests * spelling * fix: spelling * chore: PR review adjustments * add order parameter * ProposalsOrder docs * GetProposalsBriefPage supports order * add filters object * proposalsBriefPage filtering * fix: status filtering * more draft proposals filtering tests * Campaign proposals filter * update docs * expose getVisibleProposalsCount and tests * expose filters parameter * integrate proposals page with v2 queries * chore: increase time diff between proposals * chore: reduce count query tables watched when not needed * local proposals cubit * local proposal fav staus update * docs * fix: add discovery specific colors (#3637) * fix code-generator earthly target * use logger in migration + wrap in transaction * spelling * wip * rename category to categoryId for better consistency * adding vote data to ProposalBriefData * fix: dependencies * voting page + cubit integration - wip * proposals per tab selector * release completed in close * extract early return logic into function * Voting tabs selector + releasing completer on close * move spaces blocs to shell page level * fix: voting in grid view * dummy voting integration * chore: remove showComments * docs: ProposalOrDocument --------- Co-authored-by: Ryszard Schossler <[email protected]>
* docs: capture initial times * chore: remove cacheDocument * turn off logging * chore: use debugPrint instead of logger * feat: bulk documents save * batching sync * chore: move exact ref resoling to getDocumentData instead of index * fix: DocumentRepository * chore: simplify getting documents data * remove getAllDocumentsRefs and getCachedDocumentsRefs from DocumentRepository. Index is only available in DocumentDataRemoteSource * chore: update docs * simplified document index endpoint * remove randomness from LocalCatGateway * indexing by batch size * wip: filtering by types * Rework filtering refs + checking all refs if already cached in parallel * docs * update indexing.csv * update indexing csv * chore: cleanup * trailing new line * chore * move performance tab to docs * bulk saving typed docs in parallel * chore: revert hardcoded timestamp * chore: typos * split _sync into smaller functions + add documentation * little refactor * fix: analyzer * initial v2 tables * wip * wip * feat: database migration * chore: cleanup * bump batch size * cleanup * chore: remove defensive content decoding * chore: daos * spelling * saveAll * test on platform * chore: update build scripts * feat: DocumentsV2Dao methods * simple proposals pagination query * chore: create a JoinedProposalBriefEntity * rename method * feat: exclude hidden proposals * more tests * renaming and splitting logic into smaller parts * feat: per language strategy * remove CatalystDatabaseLanguage in favor of raw queries as they are easier to mange * remove Index Strategy Documentation * handle case where ref is empty * migration now includes indexes * use v2 documents table for saveAll and isCachedBulk * adds ActionType to JoinedProposalBriefEntity * adds versionIds to JoinedProposalBriefEntity * comments count * adds isFavorite to JoinedProposalBriefEntity * add template to JoinedProposalBriefEntity * adds documentsLocalMetadata table for auto updates * Update docs * use v2 proposals query for discovery most recent section * feat: simplify most recent proposals section * add proposal fav status for v2 tables * local proposal fav status update * update fav state locally for faster feedback * self review * fix tests * update times * fix: analyzer * more migration test data * clean up constructors * cleanup * fix: template tests * spelling * fix: spelling * chore: PR review adjustments * add order parameter * ProposalsOrder docs * GetProposalsBriefPage supports order * add filters object * proposalsBriefPage filtering * fix: status filtering * more draft proposals filtering tests * Campaign proposals filter * update docs * expose getVisibleProposalsCount and tests * expose filters parameter * integrate proposals page with v2 queries * chore: increase time diff between proposals * chore: reduce count query tables watched when not needed * local proposals cubit * local proposal fav staus update * docs * fix: add discovery specific colors (#3637) * fix code-generator earthly target * use logger in migration + wrap in transaction * spelling * wip * rename category to categoryId for better consistency * adding vote data to ProposalBriefData * fix: dependencies * voting page + cubit integration - wip * proposals per tab selector * release completed in close * extract early return logic into function * Voting tabs selector + releasing completer on close * move spaces blocs to shell page level * fix: voting in grid view * dummy voting integration * chore: remove showComments * docs: ProposalOrDocument * create proposals/count query .csv files * update proposals query times * category indexes * CatId parse and tryParse * create new tables for authors * update migration * run database analyze after sync * DocumentAuthors docs * simpler copyWith * cat.id keeps schema * get rid of authorCatIdWithoutUsername column * console profiler * catId -> id * times * results notes * spelling * fix: analyzer * update results (#3673) * fix: readme * remove CatalystRuntimeProfiler * catId schema -> scheme * rename * bring back profiler to SignedDocumentManager, with debounce --------- Co-authored-by: Ryszard Schossler <[email protected]>
…base_optimization_3528
…base_optimization_3528
…base_optimization_3528
* remove totalAsk and proposalsCount from campaign model * proposalsCount -> finalProposalsCount * CampaignCategoryViewModel.id -> CampaignCategoryViewModel.ref * little DiscoveryCubit state build cleanup * watch campaign total ask in discovery * smaller voting category model * new proposal campaign * safe check * proposal builder category total ask * chore: missing TODO * category details * watch proposals template total ask * finish integration of total_ask * close * remove print and add TODO * documents getLatestOf * categoryId -> categoryRef * chore: make logger private * feat: Make MultiCurrencyAmount immutable
…base_optimization_3528
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Sub-feature PR with database optimization efforts
Related Issue(s)
Resolves #3528
Description of Changes
See commit (PRs) history